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“External constituents are demanding not 
only that departments say they are doing good 
things and not only that they measure how hard 
they are trying, but also that they measure out- 
comes” (Walvoord et ah, 2000). “[Colleges] 
should be focused on the ‘value added’ of the 
student experience. In today’s society, the need 
to educate for understanding — not just grades — 
has never been more important” (Merrow, 

2004). “What counts most is what students DO 
in college, not who they are, or where they go to 
college, or what their grades are” (Edgerton as 
cited in Merrow, 2004). 

As evidenced by these quotations, the 
nature and assessment of education is changing 
significantly, and the assessment trajectory is 
away from sole reliance on the traditional per- 
spective of student grades. Technology faculty 
must respond to the changing requirements of 
student assessment and ensure that graduates 
of the program meet both the expectations and 
standards of the institution as well as those of 
other stakeholders, particularly the private sec- 
tor that typically employs those graduates. 

We are reevaluating ways in which students 
in their departments are assessed. The three-step 
“backward design” process recommended by 
Wiggins and McTighe (1998) has served as a 
conceptual framework and useful design per- 
spective. Broadly, these steps condense to (a) 
identify desired results, (b) determine accept- 
able evidence, and (c) plan learning experiences 
and instruction. Wiggins and McTighe offered a 
number of practical guidelines to the design and 
development of curricula, including the assess- 
ment process. They stated that “the backward 
design approach encourages us to think about a 
unit or course in terms of the collected assess- 
ment evidence needed to document and validate 
that the desired learning has been achieved, so 
that the course is not just content to be covered 
or a series of learning activities” (p. 12). 

We have identified a number of elements 
driving assessments at the departmental level as 
well as those that address the degree candidates’ 
needs. We have explored non-classroom-cen- 


tered assessment methods and have collected 
and analyzed preliminary data towards reaching 
the goal of more holistic assessment of student 
progression. The planning, work, and outcomes 
to date that support the learning goals and 
expected outcomes set forth by stakeholders 
follow. However, a point of reference must be 
set prior to any attempt to address the dual 
issues of student and program assessment by 
industrial technology faculty in higher educa- 
tion. For example, consistent with both our 
experiences and the recommendations outlined 
in step one of the three-step “backward design” 
process, one must have a fairly specific vision 
of the knowledge, skills, and attitudes a technol- 
ogy student should develop prior to embarking 
on his or her career before formulating an 
assessment plan. In other words, what is to 
be assessed? This might include such things 
as knowledge of technology and its associated 
processes; practical skills associated with mate- 
rials, tools, processes, and systems related to 
technology; ethics related to technology devel- 
opment and application; attitudes toward tech- 
nology; and issues impacting student learning. 

A clear understanding of the reasons for 
assessing technology students is also critical. 
These reasons may originate in basic require- 
ments to uncover information regarding stu- 
dents’ knowledge, skills, or attitudes. One may 
wish to verify that students can demonstrate 
practical technology skills and related profes- 
sional skills, or one may desire to motivate and 
enhance learning. Ultimately, it is a goal of the 
faculty to have more than just course grades to 
reflect student performance. As noted by 
Wiggins and McTighe (1998), “Too often as 
teachers, we rely on only one or two types of 
assessment, then compound that error by con- 
centrating on those aspects of the curriculum 
that are most easily tested by multiple-choice 
or short answer items” (p. 65). A well-structured 
program should include assessment by a variety 
of methods and from a more holistic perspective 
than is often currently employed. An ancillary 
benefit of a more holistic assessment may be 
a more positive student attitude toward the 
discipline. 


From the departmental perspective, 
improved student assessment allows faculty 
to do a better job evaluating and adjusting the 
technology curricula and program options (e.g., 
manufacturing and occupational safety) to meet 
educational objectives. Improved assessment 
also increases faculty credibility with other 
stakeholders in the educational process, such 
as administration, parents, and employers. Given 
a clear understanding of the overall educational 
objectives and forces driving the need for 
assessment, one may turn to questions of avail- 
able assessment strategies and benefits they 
provide for the learner and the instructor. 

We three authors formed an action research 
collaborative with the central goal being to 
develop, implement, and evaluate innovative 
forms of assessment that document student 
growth in a holistic manner. The alternative 
forms of assessment that we advance are not 
envisioned as wholesale replacements for exist- 
ing knowledge-based tests; rather, we suggest 
they supplement existing grade data with assess- 
ments targeted toward technological capability 
and problem solving. Outcomes of an improved 
assessment process allow faculty to make 
adjustments to both curricula and extracurricu- 
lar activities with greater clarity regarding the 
impact of such changes on the industrial tech- 
nology program and students. Ultimately, the 
faculty hope to accelerate the students’ learning 
more effectively and efficiently and “jumpstart” 
them into their profession. 

Technology Learning Community 

Changes to assessment processes are com- 
plicated by simultaneous changes to the pro- 
gram. For example, the context for our action 
research is a Technology “Learning Com- 
munity” (TLC) that we established within the 
industrial technology curriculum at Iowa State 
University. Learning communities are relatively 
recent additions to the landscape of American 
universities. Research by Lenning and Ebbers 
(1999) has indicated improved student retention 
and satisfaction with the academic experience 
through the use of learning communities. The 
TLC is an induction and support activity for 
freshmen and transfer students in the industrial 
technology program. The purpose of the TLC is 
to help entering students (regardless of their 
academic stage) maximize their educational 
experience and begin their professional accul- 
turation within the discipline of industrial tech- 


nology. TLC participants are organized into 
small groups of students. Each student group 
works with a peer mentor, an industrial mentor 
(an industrial technologist practicing in indus- 
try), a graduate assistant, the academic advisor, 
and industrial technology faculty members. In 
addition to more formal assessments, TLC stu- 
dents evaluate their experience each week in 
reflective summaries. Below are some com- 
ments selected from TLC students’ summaries: 

• This was a very helpful class since it 
provided to me a strong idea of what I 
can expect, and in turn what is expected 
of me. Peer mentor groups met in class 
and we exchanged phone #'s and E-mail 
addresses. 

• Last class we had speakers come in 
from SME [Society of Manufacturing 
Engineers], ASSE [American Society 
of Safety Engineers], and SPE [Society 
of Plastics Engineers] clubs who talked 
about what their organizations do and 
how to get involved in these clubs. 1 think 
this was a very good class day because 
most people are unaware of how they can 
get involved in something like this and 
what they actually do in these clubs. I am 
very interested in SME and hope to join 
next fall. I would have liked to join this 
semester but with my current schedule 
and obligations this is impossible. 

• Last week our group met with our peer 
mentor for half an hour. We talked about 
our schedules and how classes are going. 
Our peer mentor suggested that we do 
our resumes and turn them in to him the 
next time we meet. We discussed how 
important it is to go to Career Day on Feb 
19th. He wants our resumes so he can go 
over them and make possible corrections 
so we can have them with us for career 
day. We have assigned a day and time for 
our meetings, which is Wed. at 5 p.m. in 
the TLC room. 

• Last week’s class helped me to more 
fully understand the full potential that 
an Industrial Technology major could 
provide for me in the future. In our group 
meeting we talked about how to improve 
our resumes and how to sign up for ECS 
[Engineering Career Services]. 
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At the end of each semester, TLC students 
were asked to evaluate the ability of the TLC 
experience to meet the goals of the TLC initia- 
tive using excellent, good, fair, or poor ratings. 
When questioned about their experiences with 
respect to the goals of “Orientation to the indus- 
trial technology discipline and profession” and 
the “Process of developing realistic self-assess- 
ments, career goals,” 93% of the students 
reported good/excellent. The responses were 
even more positive regarding the goals of 
“Connections with faculty, other students, 
and industry professionals” and the “Process 
of introducing the variety of professional roles 
available through an industrial technology 
degree,” with 96% of the students reporting 
good/excellent experiences. 

The experiences within the introductory 
course were crafted to establish both the small 
group cohesiveness and interaction with peers 
that is so essential to effective team-based tech- 
nological problem solving that employers 
demand of industrial technology (ITEC) gradu- 
ates. Another key to the success of the TLC is 
the use of industrial mentors. ITEC students are 
very pragmatic — most have their sights set 
firmly on a career in industry. Hence, inputs 
from industrial mentors are highly valued and 
persuasive. They convey high expectations while 
demonstrating realistic practice and applications 
in industry. 

Student learning has been enhanced 
through cooperative interaction with their TLC 
group members and mentoring team. The indus- 
trial and peer mentors have increased the TLC 
students’ understanding of the discipline and the 
importance of curriculum components beyond 
what individual faculty members can accom- 
plish in the classroom. The TLC students have 
a better understanding of their personal learning 
styles and how that impacts their studying 
habits and classroom interactions. By the end 
of the semester, TLC students have generated 
or updated their resumes, started professional 
portfolios, and set the foundation of team build- 
ing and awareness of technology that they will 
continue to enhance and build upon throughout 
their academic career and beyond. Additional 
information regarding the Iowa State University 
TLC may be found in Freeman, Field, and 
Dyrenfurth (2001). 

Student Outcomes Assessment 

Desired student outcomes include enhanced 


capability with technology, increased student 
satisfaction, higher academic performance, 
refined career goals, and a greater awareness 
of one’s learning style and how to most effec- 
tively utilize that information. When students 
conclude their studies within technology pro- 
grams, it would be desirable for them to have 
indicators of capability to demonstrate academic 
proficiency, beyond just course grades. 

In order to assess student outcomes in 
a more holistic way, appropriate instruments 
must be available. Efforts are underway to 
identify and evaluate such instruments for use 
in the undergraduate industrial technology cur- 
ricula. In addition to the qualitative and quanti- 
tative analyses related to the TLC, activities 
include an evaluation of the use of Dyrenfurth ’s 
(1991) technological literacy instrument based 
on the work reported in Dyrenfurth and Kozak 
(1991), the ACT Work Keys employability skills 
exams, and the National Association of 
Industrial Technology (NAIT) certification 
exam; reviews of each student’s portfolio by 
faculty and an industrial team; and comparison 
of participant satisfaction and academic per- 
formance with other students not participating 
in TLC activities. 

A number of other key targets have been 
included as initial assessment areas. These tar- 
gets include student demographics, academic 
performance, learning styles, technological 
understanding and capability, and ethical dimen- 
sions of technology. 

With these assessments, faculty hope to 
benchmark both the initial and exit competence 
of students, document students’ progression over 
the course of their academic experiences, docu- 
ment differences among groups and types of 
students, and investigate implications arising 
from these differences for program design and 
development. The assessments are also designed 
to focus attention on the various components of 
competence (e.g., technical, managerial, founda- 
tional, personal) and to increase attention to the 
assessment process in order to strengthen its 
validity and reliability. 

The faculty has synthesized a set of 
targeted industrial technology competencies 
for students at Iowa State University through 
the efforts of individual instructors responsible 
for specific courses, the departmental curricu- 
lum committee, and other stakeholders 


(e.g., Industrial Advisory Council and NAIT). 

It would be advantageous for faculty to be able 
to collect a wide variety of demographic data 
and longitudinal data for tracking student per- 
formance, as well as data indicating perform- 
ance against desired outcomes; however, they 
recognize that constraints exist on the amount 
of data that can be collected and analyzed. 

The NAIT accredits industrial technology 
programs. Department missions are expected to 
be compatible with the approved definition of 
industrial technology. The mission, as listed in 
the 2002 NAIT Self Study Report (Department 
of Industrial Education and Technology, 2002), 
states: “The Department of Industrial Education 
and Technology at Iowa State University pre- 
pares technically oriented professionals to pro- 
vide leadership in manufacturing technology 
and occupational safety through an undergradu- 
ate industrial technology program” (p. 6.2-2). 

A second necessary, but not sufficient, con- 
dition for accreditation listed in the 2002 NAIT 
Self Study Report requires that competencies 
shall be identified that are relevant to employ- 
ment opportunities available to graduates. While 
the accreditation process serves as one driving 
force for curriculum review and outcomes 
assessment, there are others. Apart from the 
expected continuous improvement efforts of the 
faculty, the department retains an Industrial 
Advisory Council to assure that the industrial 
technology curriculum addresses the current and 
future needs of business and industry. The 
Council recommends and reviews curriculum 
and program changes that will enable the 
department to be responsive to business and 
industry (Department of Industrial Education 
and Technology, 1998). The department faculty, 
NAIT, and the Industrial Advisory Council all 
play an important role in defining and refining 
the competencies expected of industrial technol- 
ogy graduates. 

Ultimately, a reevaluation and alignment 
of course content offered in the programs was 
needed to ensure that the competencies expected 
of industrial technology graduates were realized. 
This involved a comparison of curricular con- 
tent with required NAIT objectives. Gaps and ! 
or superfluous material were identified and 
addressed. Faculty members led this effort, 
but they were not without guidance with respect 
to the process. The aforementioned three-step 
design process by Wiggins and McTighe (1998) 


and work by Kenealy and Skaar (1997) offered 
useful frameworks. Kenealy and Skaar suggest- 
ed an interesting outcomes-defined curriculum 
renewal process that has continued to influence 
this effort. Kenealy and Skaar described a multi- 
step “action planning” process that has been 
adapted so that a large number of diverse facul- 
ty and students could contribute cooperatively 
and with a sense of ownership. The needs of 
clientele groups led to the definition of major 
educational outcomes for the program, which in 
turn formed the foundation for learning experi- 
ences that would serve to meet student needs. 
Kenealy and Skaar stated that by grouping 
learning experiences, course titles and objec- 
tives are defined for a renewed curriculum. 
Subsequently, courses are defined by specific 
learner competencies, which are edited for prop- 
er sequencing. They also examined cognitive 
learning skills to ensure that upper level skills 
were represented throughout the curriculum. 

Assessment Instruments 

Assessment instruments exist that are 
appropriate and readily available and for which 
validity and reliability studies are already well 
documented. We entered into discussions with 
ACT, Inc., regarding the use of the Work Keys® 
system at the undergraduate level. This is a sys- 
tem designed to quantitatively measure certain 
employability skills. It includes job profiling 
and work-related assessments and serves a vari- 
ety of needs in the industrial and educational 
arenas. For example, the test component of the 
Work Keys system is designed to assess person- 
al skill levels in important areas of employabili- 
ty skills (ACT, 1997). There are currently eight 
tests: (a) applied mathematics, (b) applied tech- 
nology, (c) listening, (d) locating information, 

(e) observation, (f) reading for information, (g) 
teamwork, and (h) writing. ACT (1997) stated 
that educators can use the Work Keys informa- 
tion to develop appropriate curricula and 
instruction that target skills needed in the work- 
place. The Work Keys instruments have exten- 
sive reliability and validity studies completed 
(ACT, 1997) at the secondary level, but little 
information was available demonstrating that 
its use could be extended to the baccalaureate 
level. Our preliminary research results with 
undergraduates would seem to warrant addition- 
al investigation of the Work Keys system. Five 
of the eight tests were administered to under- 
graduates, including (a) applied mathematics, 

(b) applied technology, (c) locating information, 
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(d) teamwork, and (e) writing. Our primary con- 
cern was that a majority of the students would 
score at the highest scale level of the exams, 
thus diminishing the usefulness of the exams 
for assessment purposes at the baccalaureate 
level. This did not prove to be the case as only 
9.1% ( n = 33) achieved this level in the Applied 
Technology exam, 14.8% ( n = 27) achieved this 
level in the Locating Information exam, and 
no students achieved the highest level in the 
Teamwork and Writing exams (n = 26 and n = 
22, respectively). Applied Mathematics yielded 
the only exam where significant numbers of 
students (45.2%, n = 31) scored at the highest 
level. 

The ITEC program enrolls numerous trans- 
fer students from engineering. Many of these 
students have not had a great deal of academic 
success in the engineering program, but most 
seem to thrive in the industrial technology 
program. It has been posited that a mismatch 
between instructional style and learning style 
has been a leading cause of at least some of 
these students’ previous academic problems and 
that a change to a more “hands-on” curriculum 
has allowed them to flourish. While there may 
indeed be systemic differences in the instruc- 
tional approaches taken by engineering and 
industrial technology faculty, a recent study 
undertaken to investigate this question found 
that there were no statistically significant differ- 
ences in learning styles between groups of 
engineering students and industrial technology 
students at either Iowa State University or North 
Carolina A & T State University (Fazarro, 

2001). Fazarro used the Productivity Enviro- 
nmental Preference Survey to evaluate learning 
style differences (Dunn, Dunn, & Price, 1996). 

A number of other planned assessment 
instruments require some additional develop- 
ment prior to their wholesale use. Some are 
lacking sufficient reliability and validity data, 
whereas some offer only partial coverage of 
desired content areas. For example, the NAIT 
certification test currently covers only four tech- 
nology categories: (a) production planning and 
control, (b) safety, (c) quality control, and (d) 
management and supervision. There are other 
topics that fall under the technology umbrella 
such as materials and processing, industrial 
training and development, energy, instrumenta- 
tion and control, and information technology. 
Rowe (2001) suggested an updated test blue- 
print for the NAIT certification exam. Rowe 


used a modified Delphi technique to identify 
core content, subject areas, and competencies. 
Thirteen core competency areas were identified 
including (a) leadership skills for supervisors, 
(b) teamwork, (c) fundamentals of management, 
(d) safety management, (e) technical graphics/ 
CADD, (f) quality, (g) electronics, (h) human 
resource management, (i) technical writing, (j) 
written communication, (k) verbal communica- 
tion, (1) computer integrated manufacturing, and 
(m) manufacturing automation. Rowe’s findings 
indicated a need for expanding the use of writ- 
ten and verbal information, particularly with 
respect to communicating technical information. 

The current NAIT certification test is a 
cognitive, norm-based, multiple-choice test. 

It contains four 40-question subsections: pro- 
duction planning and control, quality control, 
safety, and management/supervision. Summary 
statistics, including classical difficulty and dis- 
crimination coefficients (point biserial correla- 
tion) based on a sample size of approximately 
1,200 students, are available from the authors. 
Efforts are also underway to analyze these exam 
items using item response theory (IRT) meth- 
ods. There appears to be a significant level of 
interest by NAIT-accredited programs for use 
of the certification exam in both program and 
student evaluations. 

Description and sequencing of the test con- 
struction process (preparing test specifications, 
item construction and review, detecting item 
bias, estimating reliability, etc.) are readily 
available in, for example, Crocker and Algina 
(1986). All tests used for assessment are expect- 
ed to pass a review against generally accepted 
test construction guidelines. 

Other instruments currently under consider- 
ation include Dyrenfurth’s (1991) technology 
literacy test, a survey of attitudes toward tech- 
nology (DeVries, Dugger, & Bame, 1993; Raat 
& DeVries, 1986), multiple technological prob- 
lem-solving appraisals developed in the manner 
suggested by Kimbell and Stables (1999), and 
the assessment of technology projects and activ- 
ities using group process and student individual- 
ized performance rubrics suggested by Custer, 
Valesey, and Burke (2001). 

We envisioned the deployment of the afore- 
mentioned assessment instruments across the 
department and longitudinally over the students’ 
four-year period of study. The plan would have 


the department’s faculty establish a database of 
specific course goals and objectives and then 
crosswalk each to the department outcomes. 
Existing tests and other assessments would 
be cross-indexed to these goals and objectives. 
Tables of specification would be used to identi- 
fy weighting and/or coverage gaps, inappropri- 
ate proportions, etc. 

Subsequently, innovative assessment mech- 
anisms, such as authentic assessments, portfo- 
lios, rubrics, and adaptive testing, wotdd be 
expanded to complement and/or modify the 
current assessment strategies. Some authentic 
assessment activities are currently in place in 
the industrial technology program at Iowa State 
University. Freeman and Field (1999) discussed 
assignments given to groups of safety students 
that involve job safety analyses, along with 
equipment and process reviews in labs run for 
manufacturing students. These assignments are 
identical to the tasks safety students might find 
when working in an industrial setting following 
graduation. Students have also been asked, for 
example, to develop and construct tooling for 
specific tasks in metallic materials and process- 
es labs. Many of these laboratory-based activi- 
ties offer opportunities for authentic assessment. 

Multiple-choice assessments will be subject 
to item analysis and will be recorded in the item 
database. These assessments have their place as 
tests of technological knowledge; however, 
Kimbell and Stables (1999) do not consider 
them to be valid tests of technological problem 
solving. Kimbell and Stables offer a great deal 
of insight into the development of performance 
assessment instruments for technological prob- 
lem solving and the development of assessment 
rubrics to translate performance qualities into 
numeric data for statistical analysis. 

A summary listing of the anticipated 
system development activities are shown below: 

• Conceptualize outcomes based on inputs 
from faculty, NA1T, and the Advisory 
Council. 

• Analyze objectives for each course in 
program. 

• Analyze NA1T certification examination 
table of specifications. 

• Crosswalk course outcomes to objectives. 

• Crosswalk objectives to NAIT certifica- 
tion examination table of specifications 
and exams for each course. 

• Select instruments: 


- Conduct additional literature review. 

- Review American Educational Research 
Association presentations. 

- Explore available tests or identify and 
develop appropriate new instruments. 

• Define benchmarking process. 

• Finalize assessment timing. 

• Develop assessment-sampling matrix 
that secures data, which may then be 
aggregated without imposing all tests 
on all students. 

• Bring test administration and analysis 
online through the use of off-the-shelf 
software. 

• Develop analysis plan. 

Student performance will also be recorded 
in a database and, within legal guidelines and 
pending student approval, certain elements of 
the information will be available to assist peer 
and industrial mentors in conducting discussions 
with students. Peer and industrial mentors will 
be expected to offer each student constructive 
perspectives as to how his or her performance 
is progressing against personal and professional 
standards. Assessment validation will involve 
the program’s industrial advisory council mem- 
bers so that real-world standards are not only 
employed but are also made clear to students. 

A key goal is to demonstrate student awareness 
of growth over time. Potentially confidential 
topics from a student perspective will be han- 
dled through discussions with faculty and 
faculty mentors. 

Summary 

There are clear and persistent indicators 
that changes are needed and expected in our 
system of education. In 1999, the National 
Research Council reported, “The goals and 
expectations for schooling have changed quite 
dramatically during the past century, and new 
goals suggest the need to rethink such questions 
as what is taught, how it is taught, and how stu- 
dents are assessed” (pp. 152-153). In 2001, the 
Council reviewed and expanded on trends, 
which “are changing expectations for student 
learning and the assessment of that learning” 
(National Research Council, 2001, p. 22). 

Efforts to design and implement a more holistic 
assessment of students in industrial technology 
programs are certainly timely and in keeping 
with the spirit of the recommendations by the 
National Research Council and others. Our 
work represents an initial step towards 
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capitalizing on some of the innovation that, 
while important, nevertheless reflects singular 
enhancements. Our goal was also to integrate 
several of these innovations into a more system- 
atic approach. To this end, we have conceptual- 
ized a framework for moving forward, we have 
established the feasibility of using the Work 
Keys® employability skills instrumentation, and 
we have described some necessary implementa- 
tion steps. We invite members of the profession 
to join in the challenge of implementation of 
such enhanced systems of assessment. 
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